01. Internal Components of a Reinforcement Trading Model: DQN and Rewards

Building a Reinforcement Learning Trading Agent

Key Components:

• DQN Model: Central to the agent, processes a state, and generates a Q-table ranking the quality of actions based on expected rewards.

• Reward Function: Connects actions to their potential profit-related rewards. Three actions in focus:

  • Buy: Commonly set to zero; can be adjusted to alter buying frequency.
  • Hold: Typically starting at zero; adjustable to modify holding duration.
  • Sell: Utilizes either raw profit or log returns, balancing simplicity and mathematical properties.

Reward Strategies:

• Raw Profit: Directly ties rewards to profit from a sell action.
• Log Returns: Provides additive and symmetric properties for better reward stability.

Implementation Advice:

• Start with zero rewards for buy and hold.
• Employ raw profit or log returns for selling.
• Adjust rewards iteratively to shape agent behavior.
• Ensure rewards are balanced to avoid unintended agent actions (e.g., never selling).
• Maintain additive and symmetric properties for consistency.

Supplementary Material - Neural Net Architecture

The following Medium article gives a nice overview of all the various neural net architectures and their implications for learning. When designing your DQN, this may be a useful resource for determining architecture. Remember, the only requirements for our DQN are that the input layer matches the number of features, and the output layer matches the action space size. Any other architecture changes can be made at your discretion. It is worthwhile to note that DQNs are typically feed-forward, but this is not a hard requirement.

General Architectural Design Considerations for Neural Networks